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Foreword 

The  Taehnotoov  tdentilcalion  and  Assessment  Pro  ioct  oombkad  a  numbar  of  rotated  invest  i- 


asrionatoldenttr. 

•  mUki  Mmobor  in  i  smcMc  drsUmi  mi  to  rsvlsw  rmoiurh  ind  diyiboront 
muls  and  oomrneroisSy  svslsble  products! 

p  new  tsohnotpglasdtreugh  regular  reviews  of  rosaaroh  and  dsvalopmsnt  results,  pari- 
odtesurvsys  of  spadHc  arose,  and  tdantWcatlon  of  particularly  good  axampias  of  the 
appfleadon  of  mmcMc  technologies;  and 

•  requesmeras  ror  new  Mcnnorogy  inrougn  oonunuing  nuoat  or  sonwara  oavaioprnern 
needs  wWn  the  DoO,  and  ease  studtos  of  both  successful  and  unsuccessful  pro)- 


HMMpjf  iBWHWM  wh^iwi  unusnovong  mo  lOtwepw  oiviiopniini  ptuOiMi  owiinwiiny 


now  tschnologlss.  A— easmsnt  scbrtdaa  of  the  project  focused  on  core  technology  aroas  for 
software  engineering  snvfrofimiJrits. 

TMs  roport  is  one  of  a  oodse  of  survey  nports.  I  is  not  Mended  to  provide  an  exhaustive 

w  sD^aiDs  ^wvwv  w  w  wei  w  i^f^^8*ns  nwwwWjy*  mniii  i  s  vunoiQ 

■n  noniMW  vvvm  oi  wiv  wawogy  Hmvyva.  mm  MVijfi  fm  oonoucisa  in  ot  itoD 

wdesriyldM. 

Members  of  die  profsot  rooogntoed  dial  more  gsnsral  technois*®?  surveys  have  been  conducted 
by  odwr  bwoadgotoro.  The  project  add  not  abempt  to  duplcals  thoee  surveys,  but  tocueed  on 
points  not  addressed  In  dwe*  surveys.  The  goal  In  conducting  the  surveys  was  not  to  describe 


unique  to  software  engine  ertng  atMwnmswta.  The  objscdvt  In  prsesntlng  dwee  reports  is  to 
provide  an  ovenrfew  of  die  tschnotoglss  diet  sroooro  to  developing  sodwaro  engineering  environ- 
ments. 


1.  Introduction 

One  of  the  core  technology  areas  In  which  project  members  were  Ms  reeled  Is  dMributed  sys- 

Aabm*  Vkla  ayakAal  m  m  mi  lit  iKa  IwmImpIwwI  iwa  bkkMhkaMi  lee  jiaa^udaw  <S|^adv aail  aua 

wmm  isomoogy.  itw  npon  surveys  ins  ncnracs  mmi  imwio  si  CMQrany  awiMio  sys* 
MSt  wsi  pvwMir  snjJnasw  on  moss  ispscis  ms  wvu  suiissv  •npnwnnp  visiuiiisivi. 

Economics  Is  ths  driving  teres  behind  the  proderetion  of  dfrtrtbulsd  systems.  Morirstattons  era  a 
OBSt-aHacdva  wav  of  orovfcSno  fy>mrnrtinfi  oower  to  IndMduais  Local  area  networks  are  a  cost* 

wroreirewave  wwy  see  pwwvw^iii  vwwwi  er  wnwvwwim>  ewnrea  w>vw  i»wi*vw*f*w  ei  iwrei 

ariacffva  wav  of  aharino  aooass  to  more  exnenaive  and  laas  twi  wnriv  uaad  raaouroas  dee  laser 
printers  end  lerge  dWe,  es  wei  as  a  means  tor  users  to  share  Information  . 


Sew  aA^^d  mmSabbb  Immmbub#  IRmbb  Ib  a  A*  MriBBB*eAei  I^b  a^b^b^S  Rbbm  IhAambm 

Si  QSnDInwU  sysisms,  fWSwM|  WiSTO  IS  8  fUnOSTreMSH  OwlOIOn^f  DS8RrS8n  ms  iiOOO  iw  dVvQri* 
don  (to  achieve  sharing)  and  ths  naed  for  autonomy  (to  control  one's  local  environment).  Many  of 


die  technical  solutions  prannUd  here  can  to  evaluated  In  terms  of  how  they  balance  these  two 


OkMuM  systems  have  i  number  ol  Ml  known  potential  benefit: 


rsnormanos.  rarassssm  cen  oe  useo  e>  penorm  jods  more  qurciuy. 

•  AvatobMty:  Redundancy  cen  be  ueed  to  provide  non-Mop  service. 

To  mmm  tienelli  In  nredice.  however  reauires  eohiHons  for  e  number  of  dfificul  tech- 

e  ee  eee^w^e  ww*  er  *  iwww* w*  |  wo  ®o  row  o  r  or*  roror  or  ^orrewpo  *oor  r 

ntetf  probfemt.  Also,  there  tie  trade-off  reMonehipe  between,  for  example,  using  a  dtetributed 
system  for  Inoroossd  performance  emus  using  R  for  increased  avalaMRy.  The  following  sec- 
dons  rfiscuss  some  of  the  important  technologies  and  issues  invoked  in  dtetributed  systems. 
Heferenoee  to  appropriate  surveys  are  included  in  the  dtecussion,  but  two  general  references  are 
appropriate  here.  First,  the  book  edRed  by  Lampoon  at  at.  [18]  is  an  excelent  overview  of  many 
aspects  of  (attributed  systems.  Second,  Tanenbaum's  book  on  computer  networks  [36]  is  prob¬ 
ably  the  best  starting  point  for  more  information  on  networks  and  protocols. 

2.  Hardware  Tachnology 

COUnOfTiC  1190*9  W  ■  na^Of  iwalOn  lOf  tnv  pfOmOrKIOn  Ov  QKIiVlnwO  vyilwinai  riOCw8SOrPi 

vmmory,  ino  imgnitic  ana  opucii  onks  vo  suvncwntiy  mixponBivo  m  mow  m  organizmon  to 
deploy  a  workstation  in  every  office,  In  addlion  to  supporting  a  machine  room  with  mainframes 
and  «e  servers.  Current  workstations  typicaly  have  from  1  to  10  MIPS  (miHon  instructions  per 
second)  of  processing  power,  1  to  10  megabytes  of  memory,  20  to  200  megabytes  of  dtek 
storage,  and  ooet  less  than  20,000  dolars. 

a  wwy  Qi  novwovK  ivcnnovogw!  mb  mhuxi  nor  irMfwnnocung  univ  mmpofwiw.  a  hrimc* 
don  a  commonly  made  between  local  area  and  long-haul  networks.  For  both  physical  and  ad- 
miniatrattve  reaeons,  local  area  networks  (LANs)  are  typicaly  used  within,  and  managed  by,  a 
single  organization.  Currently,  most  LANs  are  constructed  from  coaxial  cable  or  fiber  optics,  with 
bandwtdths  ranging  from  10  to  100  ml  Son  bis  per  second  and  lengths  on  the  order  of  i 
klometer.  The  interconnection  topologies  of  such  networks  include  bus,  ring,  tree,  and  star  struc¬ 
tures. 

Long-haul  networks,  on  the  other  hand,  can  span  continents,  and  are  usually  managed  by  com¬ 
panies  or  government  agencies  for  use  by  others.  The  technologies  used  for  long-haul  networks 
Include  telephone  Ines  and  sat  el  Re  Inks.  Long-haul  networks  use  one  or  more  of  the  following 
switching  techniques: 

•  CJrcut  twHchina 

•  Message  switching 

•  Packet  swlching 


Circufi  swlching  is  used  in  the  telephone  system.  In  this  scheme,  a  route  through  the  network  is 
establshed  before  any  data  is  sent.  Since  communication  between  computers  tends  to  come  in 


i 

I 


burets,  ckcul  swtchlng  does  not  provide  good  utHzsiion  of  svaiable  barxkridths,  end  the  time 
required  to  pre-atocate  a  drew*  may  be  unacceptably  long. 

Message  switched  and  packet-switched  networks  are  also  cased  store-and-forward  networks  be¬ 
cause  data  is  independently  routed  from  one  switching  node  to  another.  For  reasons  of  reliability, 
the  network  topology  of  a  store-and-forwaid  network  should  be  connected  redundantly,  so  that 
there  are  several  paths  or  routes  between  any  two  nodes. 

In  message  awlching,  individual  manages  are  routed  from  one  switching  node  to  another.  This 
elminates  the  long  set-up  time  associated  wth  circuit  switching  and  provides  better  utHzation  of 
bandwidths.  The  dfcadvantage  of  this  approach  is  that  the  variable  size  of  messages  makes  it 
dHicuRto  aRocate  node  buffering  resourcn  efficiently. 

In  packet  swtching,  messages  are  first  broken  up  into  fixed-size  packets,  which  are  then  in- 
dMdualy  routed  through  the  network  and  reassembled  at  the  destination.  Dtfferent  packets  of 
the  same  message  may  be  routed  along  afferent  paths,  and  hence  may  arrive  out  of  order. 
Higher  level  protocols  are  used  to  handle  out-of-order  packets.  Since  alt  packets  are  of  the  same 
(relatively  small)  size,  buffering  at  intermediate  nodes  is  sknpVied.  The  Arpanet  is  an  example 
of  a  long-haul  packet-swRched  network. 


3.  Internetworks 

Because  of  their  low  cost,  workstations  tend  to  proMerate  in  organizations,  and  the  need  for  LANs 
tends  to  grow  as  well.  Large  organizations  are  soon  faced  wth  the  necessity  of  connecting 
several  LANs  through  a  structure  caled  an  internetwork.  In  fact,  some  of  the  components  of  an 
internetwork  can  be  long-haul  networks.  In  the  oarpa  Internet,  for  example,  the  long-haul 
Arpanet  is  connected  wth  hundreds  of  LANs  at  universities  and  research  laboratories.  The 
same  store-and-forward  approach  can  be  used  hi  an  internetwork,  viewing  the  internet  gateways 
as  the  packet  switches  of  a  single  larger  network. 


4.  Protocols 

Protocols  are  used  to  proride  virtual  communication  services  with  properties  different  from  (and 
typically  at  a  higher  level  than)  those  provided  by  the  physical  network.  This  leads  naturally  to  a 
layered  model  of  protocols,  such  as  the  one  that  has  been  standardized  in  the  ISO  Reference 
Model  for  Open  Systems  Interconnection  [17]. 

Further  rfscussion  about  protocols  may  be  found  in  the  survey  article  by  Tanenbaum  [37]. 

4.1.  ISO  Reference  Modol 

The  ISO  Reference  Model  consists  of  the  foSowing  seven  layers: 

1.  Physical  layer:  low-level  communication  of  Ms 

2.  Data  Ink  layer:  framing,  checksumming 
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o.  nviwOfK  iiyor.  nwmsiwOfK  naressvig  ana  reuimQ 

4.  Transport  layer  reliable  communication,  host-to-hoet  addressing 

5.  Session  layer:  connection  management,  process-to-process  addressing 

6.  Presentation  layer,  data  formatting,  encryption,  compression 

7.  Appfication  layer  user  programs 

The  ISO  Reference  Model  does  not  match  all  protocol  architectures  perfectly.  In  the  DoD  Internet 
family  of  protocols,  tor  example,  the  IP  [26]  and  TCP  [27]  protocols  provide  functionality  that 
ranges  from  the  data  link  layer  to  the  session  layer. 

4.2.  Transport  Protocols 

Communication  services  can  be  characterized  by  a  number  of  attributes: 

•  The  need  for  a  connection  establishment  protocol  before  communication  can  occur 

•  The  number  of  communicating  entities 

•  ReiabMy  of  data  delvery 

•  dent  interface  (messages  or  stream  abstraction) 

•  Fixed  or  variable  length  of  messages 

The  foSowing  is  a  brief  characterization  of  a  number  of  transport  protocols  according  to  the  above 
attributes. 

•  Datagram  protocol:  connectionless,  unreliable  delivery,  fixed-size  packets.  Ex¬ 
amples  include  the  Xerox  PARC  PUP  protocol,  the  DoD  Internet  Protocol  (IP),  and 
fie  DoD  User  Datagram  Protocol  (UDP). 

•  Byte  stream  protocol:  connection-based,  reliable  delvery,  stream  abstraction.  Ex¬ 
amples  include  the  Xerox  PARC  byte  stream  protocol  (BSP)  and  the  DoD  trans¬ 
mission  control  protocol  (TCP). 

•  Message  protoool:  connectionless,  ratable  delivery,  variable-length  messages.  Ex¬ 
amples  include  the  protocol  used  by  the  Spice  system  [29]. 

•  Request/response  protocol:  connectionless,  ratable  delivery,  variable-length  alter¬ 
nating  request/response  messages.  Examples  are  described  by  Birred  and 
Ne!aon[6]. 

A  recent  topic  of  research  has  been  the  incorporation  of  many-to-many  communication  semantics 
Into  various  transport  protocols  [1,8, »].  New  protocols  In  each  of  the  above  classes  win  Ifcely  be 
extended  wth  many-to-many  semantics. 

4.3.  Highsr-Lsvtl  Protocols 

Higher-level  protocols,  those  implemented  at  the  session  layer  or  higher  in  the  ISO  model,  are 
correspondingly  harder  to  characterize.  Examples  from  the  DoD  Internet  family  include  the  Tel¬ 
net  network  terminal  protocol,  the  ffle  transfer  protoool  (FTP),  and  the  mal  delivery  protocol 
(SMTP).  Other  areas  of  research  include  protocols  tor  graphics,  window  managers,  voloe,  multi- 
medta  messages,  bootstrap  loading,  remote  debugging  and  monitoring,  and  remote  procedure 
cal.  dtocussed  more  fully  below. 
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5.  Heterogeneity 

Currently,  there  is  no  single  standard  machine  arcNtecture,  operating  system,  programming  lan¬ 
guage,  or  programming  environment,  and  such  standards  are  not  Nkety  to  appear  in  the  near 
future.  As  a  result,  organizations  find  themselves  faced  with  the  problem  of  integrating  a  hetero¬ 
geneous  colection  of  such  resources.  As  evidenced  by  a  recent  workshop  in  East  sound, 
Washington,  that  was  devoted  solely  to  the  problems  of  heterogeneity,  and  by  current  research 
projects  in  heterogeneity  at  institutions  such  as  CMU  and  MIT,  this  is  the  key  problem  in  distri¬ 
buted  systems  today  124]. 

The  most  important  pitfall  to  avoid  in  a  heterogeneous  system  is  the  lowest  oommon  denominator 
effect.  This  occurs  when  interfaces  are  only  defined  for  those  operations  that  are  supported  by  all 
components  in  the  system.  As  the  number  of  heterogeneous  components  increases,  this  set  of 
common  operations  may  approach  the  empty  set. 

A  number  of  techniques  can  be  used  to  avoid  the  lowest  oommon  denominator  effect.  One 
technique  is  a  common  data  representation  protocol,  in  which  al  oommunicating  components 
translate  their  interactions  into  a  standard  externa)  representation.  As  described  below,  this  can 
be  handled  automatically  in  remote  procedure  cal  systems  through  the  use  of  a  stub  generator. 
The  main  difficulty  wth  this  technique  is  that  the  representation  protocol  Iseff  suffers  from  the 
lowest  common  denominator  effect.  The  advantage,  however,  is  that  such  protocols  are  flexble 
since  they  are  capable  of  representing  arbitrary  programming  language  data  types  fee  arrays  and 
records.  DeSchon  surveys  a  number  of  data  representation  standards  [10]. 

Another  technique  is  caled  option  negotiation  [34],  in  which  each  pair  of  communicating  parties 
negotiates  which  protoool  options  they  wW  support.  This  approach  alows  each  pair  to  commu¬ 
nicate  wth  maximal  functionality.  The  option  negotiation  approach  is  applcabie  at  many  levels  in 
a  heterogeneous  dtetributed  system. 

The  data  representation  protoool  and  option  negotiation  techniques  can  be  successful  com¬ 
bined.  FOr  example,  the  remote  procedure  cal  system  at  the  DEC  Systems  Research  Center 
uees  negotiation  at  bindkig  time  to  decide  between  two  possbie  data  representation  protocols. 

A  third  and  somewhat  ad  hoc  approach  to  coping  wth  heterogeneity  is  the  proxy  technique.  A 
proxy  is  a  specialized  agent  in  a  remote  environment  whose  purpose  is  to  provide  an  Interlace  to 
that  environment  that  is  more  compatible  with  other  components  of  the  system.  This  approach 
was  first  used-  in  remote  Job  entry  (RJE)  systems  to  access  batch  facilities  from  timesharing 
systems.  R  has  been  used  successfully  in  the  Locus  system  [25]  to  Integrate  IBM  mainframes 
transparently  into  a  distributed  Unix1  environment. 


iijmx  it  a  ftoftsMd  fridivniffc  of  M  ijboraftorioi 


6.  Models  of  Distributed  Programs 

Afthough  transparency  is  desirable  at  the  highest  levels  of  a  dtetributed  system,  at  some  lower 
level  the  (act  that  the  system  is  dtetributed  must  be  made  available  to  the  programmer.  How  this 

is  done  to  largely  determined  by  the  model  of  dtetributed  programs  that  the  systems  designer 

— •  — *— 

MOptS. 

One  of  the  most  wel  known  approaches,  developed  at  Xerox  PARC  in  the  1970s,  is  called  the 
ckent/server  model.  The  computing  environment  is  assumed  to  consist  of  personal  workstations 
and  a  colectlon  of  shared  network  services  implemented  by  server  machines.  Such  services 
might  include  We  storage  (discussed  more  fuRy  below),  printing,  and  electronic  mail.  The  pro¬ 
grams  running  on  the  user's  workstation  are  viewed  as  cNents  of  these  servers.  The  client/server 
model  is  a  simple  extension  of  the  application  program/operating  system  model  famliar  in  central¬ 
ized  timesharing  systems.  It  is  flextoie  because  new  services  are  easily  added,  and  it  supports  a 
heterogeneous  environment  wel:  'Black  boxes"  can  be  used  as  servers  as  long  as  some  inter¬ 
lace  can  be  constructed  on  the  dent  side.  A  dteadvantage  of  the  client/server  model  is  that  it 
does  not  support  load  balancing  or  multi- machine  parallel  apptications,  although  such  program 
structures  can  be  shoe-homed  into  this  model  by  using  a  pool  of  "compute  servers." 

Some  of  these  deficiencies  are  remedied  in  the  network  operating  system  (NOS)  model.  In  this 
model,  a  transparent  interface  to  al  network  resources  is  presented  to  the  applications  program¬ 
mer.  not  Just  at  the  user  interface  level.  The  Locus  system  at  UCLA  [25]  and  the  Spice  system  at 
CMU  [29]  are  successful  examples  of  systems  that  follow  this  model.  A  major  dteadvantage  of  the 
network  operating  system  model  is  Its  dKHcu&y  in  accommodating  heterogeneity  (in  the  form  of 
black  boxes)  because  k  assumes  that  a  common  software  interface  can  be  installed  on  an  the 
network  resources. 

7.  Operating  System  Issues 

This  section  briefly  describes  a  number  of  operating  system  features  that  are  particularly  impor¬ 
tant  for  supporting  distributed  systems. 

A  message-based  operating  system  consists  of  an  efficient  kernel  implementation  of  processes, 
virtual  memory,  and  inter-process  communication,  together  wfth  a  set  of  server  processes  provid¬ 
ing  conventional  operating  system  services  such  as  devioe  drivers  and  fie  systems.  The  Accent 
kernel  Is  a  prime  totempie  of  a  message-based  system  [29]. 

Message-based  kernels  alow  inter-process  communication  to  be  extended  over  the  network  in  a 
simple  and  transparent  fashion.  The  key  is  the  notion  of  intermediary  processes  that  intercept 
remotely  destined  messages  and  perform  the  appropriate  torwardtog. 

There  is  growing  agreement  that  a  Kghtweight  process  mechanism  to  essential  to  support  com¬ 
monly  used  dtetributed  program  structures.  A  number  of  Ightweight  processes  can  share  a  single 
address  space;  this  slows  the  construction  of  servers,  for  example,  that  correctly  handle  concur¬ 
rent  Incoming  requests.  The  lack  of  such  lightweight  processes  has  been  a  weak  point  of  Unix 
and  a  number  of  message-based  operating  systems. 


A  process  migration  facttty  allows  a  running  process  to  be  moved  from  one  machine  to  another. 
Such  a  facttty  is  a  valuable  mechanism  tor  implementing  load  balancing  poRcies,  whereby  jobs 
we  moved  off  heavfly  loaded  machines  and  onto  lightly  loaded  ones.  Variants  of  process  migra¬ 
tion  can  be  used  to  increase  fault  tolerance  by  checkpointing  process  state.  Process  migration  is 
greatly  simpBfied  in  message-based  operating  systems  [28]. 

A  simpler  form  of  toad  balancing  can  be  accomplished  at  task  creation  time  by  starting  the  task 
on  a  lightly  loaded  processor.  Further  experience  is  needed  to  determine  whether  the  full  power 
of  process  migration  is  necessary. 

Workstation  technology  has  advanced  to  the  point  where  most  new  high-end  workstations  are 
multiprocessors  with  approximately  10  processors.  Operating  system  support  for  multiproces¬ 
sors,  and  in  particular  for  efficient  execution  of  parallel  programs,  will  be  an  increasingly  important 
requirement. 

Finally,  Unix  compat bitty  is  often  a  practical  necessity.  The  wide  variety  of  software  tools  avail¬ 
able  under  Unix  would  be  prohtoitiveiy  expensive  to  port  to  an  incompatble  environment. 

Many  of  the  features  mentioned  in  this  section  have  been  included  in  the  design  and  implemen¬ 
tation  of  the  MACH-1  operating  system  at  CMU  [3],  a  kernel  and  programming  environment  that 
will  probably  serve  as  the  new  foundation  for  OARPA-eponsored  research  in  strategic  computing. 

8.  Programming  Language  Issues 

One  approach  to  integrating  distributed  programming  primitives  into  the  programming  environ¬ 
ment  is  to  incorporate  them  into  the  programming  language  Itself.  This  approach  can  be  accom¬ 
plished  in  two  ways:  the  mechanisms  can  be  butt  into  the  language,  or  they  can  be  provided 
externally. 

CSP  [16]  and  Ada  [12]  are  examples  of  languages  wfth  butt-in  communication  primitives.  This 
approach  extends  the  benefits  of  strong  typing  to  rfistrbuted  programs  because  the  language  is 
the  only  interface  to  the  communication  mechanism.  Unfortunately,  most  languages  of  this  type 
ignore  the  problem  of  heterogeneous  environments.  As  discussed  previously,  in  order  to  cope 
with  heterogeneity,  some  common  data  representation  protocol  or  negotiation  scheme  must  be 
used  among  the  language  implementations  on  different  machines.  Without  a  language-defined 
standard,  programs  produced  by  different  compilers  are  unRkely  to  be  able  to  communicate.  Ada 
provides  only  a  partial  solution  to  this  problem  in  the  form  of  pragma  statements  that  allow  control 
over  the  representation  of  data  types. 

In  message-based  operating  systems,  primitives  for  message  communication  are  typically  inte¬ 
grated  into  the  programming  language  in  the  form  of  a  subroutine  Ibrary.  Again,  little  support  for 
heterogeneity  has  been  provided.  Issues  of  data  representation  and  type  safety  are  usually  the 
responsbfllty  of  the  programmer. 

Remote  procedure  can  (RPC)  systems  represent  a  compromise  between  the  built-in  and  the 


external  approach.  By  using  a  stub  generator,  the  remote  procedure  caU  mechanism  can  be 
ctosely  coupled  to,  yet  separate  from,  the  oompiler.  This  approach  is  described  in  more  detail  in 
the  next  section. 

9.  Remote  Procedure  Call 

Remote  procedure  call  is  a  combined  protocol-level  and  language-level  mechanism  for  construct¬ 
ing  dtetributed  programs.  A  remote  procedure  call  mechanism  allows  a  programmer  to  write  a 
distributed  program  in  the  same  way  one  writes  a  single-machine  program:  using  procedure  calls 
in  one's  favorite  programming  language.  Remote  procedure  can  meshes  well  with  both  the 
dent/server  and  network  operating  system  models. 

The  language-level  integration  of  remote  procedure  call  into  a  conventional  programming  lan¬ 
guage  is  typically  accomplished  by  the  use  of  a  stub  generator,  a  specialized  compiler  that  trans¬ 
lates  a  module  interface  into  stub  procedures  for  the  client  and  server  halves  of  a  remote  inter¬ 
face.  The  stub  procedures  handle  the  details  of  representing  the  data  types  of  the  programming 
language  in  an  external  form  when  they  are  sent  in  messages,  and  the  conversion  to  and  from 
the  internal  form.  The  stub  procedures  also  Interface  with  the  lower  level  request/response 
protocol  used  to  exchange  the  call  and  return  messages. 

The  stub  generator  approach  has  a  number  of  advantages: 

•  The  stub  generator  manipulates  source-level  programs,  so  strong  typing  can  be  pro¬ 
vided. 

•  The  stub  generator  is  separate  from  the  oompiler,  so  the  same  stub  generator  can  be 
used  with  any  oompiler  for  that  language. 

•  The  stub  generator  is  a  natural  place  to  “hide”  knowledge  about  the  external  repre¬ 
sentation  protocols  and/or  negotiation  schemes  used  between  heterogeneous 
machines. 

To  invoke  a  remote  procedure,  the  client  stub  buHds  a  call  message  containing  the  name  of  the 
procedure  to  be  invoked  and  the  external  representation  of  its  arguments.  The  client  sends  the 
cal  message  to  the  server  machine,  where  It  is  interpreted  by  the  server  stub.  The  arguments 
are  converted  to  their  internal  representation  and  are  passed  to  the  named  procedure.  When  the 
procedure  returns,  its  results  are  externalized  In  a  return  message  and  sent  back  to  the  dent. 
Finally,  the  dent  stub  oonverts  the  results  back  into  internal  form  and  returns  them  to  the  dent 
program. 

Nelson  gives  a  comprehensive  treatment  of  remote  procedure  can  in  Ns  thesis  [23].  Birred  and 
Nelson  describe  the  transport  protocol  and  binding  mechanisms  used  in  an  implementation  of 
RPC  at  Xerox  PARC  [6]. 


9.1.  Advantages  of  Remote  Procedure  Call 

The  single  biggest  advantage  of  remote  procedure  cal  is  that  it  makes  writing  distributed  pro¬ 
grams  almost  as  easy  as  writing  single-machine  programs.  The  same  software  development 
methodologies  that  work  well  for  centralized  systems,  such  as  the  use  of  modularity,  abstract 
data  types,  and  stepwise  refinement,  continue  to  work  just  as  well  when  extended  with  remote 
procedure  cal. 

9.2.  Disadvantages  of  Ramota  Procedure  Call 

Although  remote  procedure  call  has  become  extremely  popular,  I  is  not  a  panacea.  In  particular, 
R  is  not  suitable  for  the  transfer  of  large  amounts  of  data,  or  for  communication  over  high-latency 
media.  Special  buk  data  transfer  protocols  are  preferred  in  such  cases. 

One  common  criticism  of  remote  procedure  call,  namely  that  the  synchronous  nature  of  remote 
procedure  cal  does  not  allow  any  parallelism,  is  really  not  a  problem,  in  fact,  remote  procedure 
call  neither  helps  nor  hinders  paraleUsm.  The  above  criticism  is  usually  accompanied  by  an 
argument  in  favor  of  non-blocking  remote  cads,  where  the  application  can  either  poll  for  the  return 
value  or  have  R  delivered  asynchronously.  Such  features  are  actually  a  poor  man's  substitute  for 
Ightweight  processes,  and  are  only  desirable  In  environments  where  processes  are  heavyweight 
and  expensive.  If  Hghtweight  processes  are  wel  supported  in  the  programming  language  and 
environment,  they  become  the  natural  means  of  achieving  parallelism  in  conjunction  with  remote 
procedure  caH.  N  not.  poMng  or  asynchronous  delivery  mechanisms  can  be  simulated  with 
remote  procedure  can,  but  use  of  such  features  can  result  in  rather  convoluted  programs.  For  the 
most  effective  match,  systems  should  support  both  remote  procedure  call  and  lightweight  proc¬ 
esses. 

10.  Software  Tools  for  Distributed  Environments 

Making  software  fools  function  transparently  in  a  distributed  environment  often  requires  substan¬ 
tial  effort..  Consider  some  of  the  tools  that  have  become  standard  equipment  in  centralized 
environments: 

•  Compilers 

•  Linkers 

•  Debuggers 

•  Profiling  tools 

•  Version  control  and  system  configuration  tools 

A  number  of  issues  must  be  addressed  when  extending  these  tools  to  distributed  environments. 

Programming  language  compilers  and  Interpreters  must  be  integrated  wRh  communication 
facMties  such  as  message  primkives  or  remote  procedure  cad.  The  software  engineering  issues 
are  compicated  by  machine  dependencies,  language  dependencies,  and  compiler  dependencies, 
any  one  of  which  can  effect  the  representation  of  programming  language  data  types  in  messages. 

Debuggers  must  be  extended  to  alow  single-stepping  across  machine  boundaries  when  following 


a  chain  of  remote  procedure  cafe.  It  should  be  poeei)le  to  set  breakpoints  in  remote  modules 
and  to  trace  the  flow  of  control  o<  a  distributed  program.  An  advantaoe  of  messaoe-based  operat¬ 
ing  systems  for  dttributed  debugging  is  the  abttty  to  encapsulate  the  entire  environment  of  a 
process,  since  al  of  its  interactions  occur  via  messages. 

Profiling  tools  provide  the  programmer  with  histograms  of  where  time  is  spent  in  a  program.  This 
alows  the  programmer  to  detect  bottlenecks  and  to  apply  optimizations  where  they  wW  do  the 
most  good.  In  the  dtetributed  case,  profiling  must  work  correctly  when  portions  of  the  program 
execute  at  remote  nodes. 

Version  control  and  system  configuration  is  a  particularly  difficult  problem  in  a  distributed  environ¬ 
ment.  Schmidt  describes  a  variety  of  techniques  for  maintaining  consistent  releases  of  large 
software  systems  in  the  Xerox  PARC  environment  [31).  Shared  fie  servers,  discussed  below,  are 
essential  to  the  success  of  such  a  scheme. 


11.  Security 

A  distributed  environment  raises  a  number  of  security  issues.  First,  the  broadcast  nature  of  most 
local  area  networks  makes  them  particularly  vulnerable  to  eavesdropping.  Anyone  with  a  per¬ 
sonal  workstation  on  an  Ethernet  can  easily  monitor  al  network  traffic.  Secondly,  the  lack  of 
control  over  the  software  run  in  an  individual  workstation  mokes  masquerades,  replays,  and 
simlar  active  threats  posstoie. 

These  problems  are  solved  in  single-machine  or  centralized  environments  by  physical  security: 
locked  machine  rooms  and  protected  terminal  Hnes.  Unfortunately,  the  decentraMzed  nature  of 
distributed  systems  precludes  such  measures.  Logical  rather  than  physical  schemes  must  be 
used  instead. 

The  simplest  problem  to  solve  is  that  of  eavesdropping.  The  solution  uses  encryption:  two  per¬ 
sons  wishing  to  communicate  do  so  by  encrypting  an  their  messages  with  a  secret  key  known 
only  to  them.  This  effectively  constructs  a  secure  private  communication  channel  on  top  of  the 
underlying  insecure  pubHc  channel.  The  Data  Encryption  Standard  (DES)  can  be  used  tor  secret- 
key  encryption  and  decryption  (21].  Hardware  implementations  of  DES  are  avalabie  and  should 
be  included  in  new  workstations 

More  elaborate  encryption-based  schemes  can  be  used  to  solve  the  authentication  problem,  in 
order  to  prevent  masquerades  and  similar  active  threats  (11, 22].  In  such  a  scheme,  a  person 
can  securely  ktentVy  Nmsef  to  another  person  by  obtaining  from  a  mutually  trusted  authen¬ 
tication  service  an  ^proof  of  Ideotfy"  that  is  unable  to  be  forged.  Birred  has  described  a  compre¬ 
hensive  scheme  that  provides  both  privacy  and  authentication  for  remote  procedure  cafe  (7]. 

The  encrypt  ion-based  schemes  that  have  been  proposed  in  the  Iterators  do  not  afford  much 
protection  against  denial-of-senrice  attacks.  R  has  been  observed  that  passive  threats  are  dffficuR 
to  defect  but  easy  to  prevent,  whle  active  threats  are  easy  to  defect  but  ddficud  to  prevent. 


12.  Distributed  Fils  Systems 

Distributed  Ms  systems  hove  mors  impact  on  programmlnB  environments  than  any  other  aspect 
of  dtetributed  systems.  A  good  dtacussion  of  fito  servers  and  dtetributed  Me  systems  may  be 
found  in  the  survey  article  by  Svobodova  [35]. 

12.1.  Filss  and  Dirsctortes 

Rtes  are  the  primary  means  of  storing  and  sharing  bng-iived  information  in  computer  systems. 
Ffle  systems  may  impose  stnjcture  on  the  contents  of  ties  (index  or  record  structures  or  file 
types)  or  may  treat  the  contents  merely  as  sequences  of  bytes.  This  report  takes  the  latter 
approach  and  views  a  He  as  a  sequence  of  uninterpreted  bytes;  any  structure  imposed  on  file 
oontents  is  viewed  as  a  logicaly  higher  level.  A  common  approach  is  to  deal  only  in  machine- 
eensfcte  unique  identifiers  at  the  file  system  level. 

A  separate  concept,  often  lumped  together  wth  the  file  system,  is  the  directory  system,  which 
provides  a  mapping  from  user-sensfcie  names  to  ffle  identifiers.  Directories  may  themselves  be 
implemented  as  fles  containing  name/identifier  pahs.  The  dbectory  system  implements  creation, 
deletion,  lookup,  and  enumeration  of  nama/tdentifier  pairs.  Additional  functions  may  include  ex¬ 
pansion  of  patterns  containing  wldcard  characters. 

The  dbodory  system  is  responsible  for  any  structuring  of  file  names.  A  common  approach  is  a 
tree  structured  dboctory  system,  bi  which  the  ful  name  of  a  ffle  is  a  path  name  consisting  of  a 
sequenoe  of  components  starting  with  the  root  dboctory  of  the  tree.  For  example,  to  the  Unix 
dboctory  system  (probably  the  most  common  tree-structured  system)  the  path  name 
AM0boopqper.tex  denotes  the  ffle  found  by  starting  at  the  root  dbectory  (the  leftmost  “/"),  consult- 
tog  the  dboctory  usrto  And  the  dbectory  ecc;  which  to  turn  contains  the  entry  paper.tex.  to  the 
Unoc  system,  only  the  V  is  interpreted  by  the  dbectory  system;  ffle  extensions  such  as  .fox  are 
purely  convention.  Other  dbectory  systems  provide  more  support  for,  and  often  more  restrictions 
on,  the  use  of  ffle  extensions.  Another  feature  of  directory  systems  that  is  missing  from  Unix  is 
the  provision  of  mutipte  versions  of  ffles.  Versions  are  typically  specified  through  additional  file 
name  syntax,  and  ffle  operations  typfcaty  use  (Afferent  default  versions  K  none  is  specified.  For 
example,  opening  a  ffle  for  reading  would  defaut  to  the  most  recent  version,  while  deleting  a  file 
would  defaufl  to  the  oldest  version. 

A  Anal  component  is  the  protection  system,  often  subsumed  by  the  dbectory  system.  For  ex¬ 
ample,  the  dbectory  system  can  alow  access  control  lists  to  be  associated  wth  each  dbectory 
entry,  and  can  provide  default  acoess  controls  through  an  inheritance  mechanism.  Note  that  an 
access  control  mechanism  presupposes  some  method  of  securely  identlytog  people,  to  a  distri¬ 
buted  environment,  this  can  be  accomplished  wth  an  authentication  service  as  outlined  above. 

122.  Sharing  Fites  In  a  Distributed  System 

The  ease  wth  which  files  can  be  shared  to  a  (Sstributed  system  is  a  good  measure  of  the  overall 
suocess  of  the  system.  Several  approaches  are  possfcle.  The  lowest  level  technique  is  the  disk 
server.  A  dtek  server  can  be  viewed  as  a  multiported  disk  controfler  whose  UO  bus  is  the  net¬ 
work.  This  approach  requires  minimal  changes  to  the  operating  system  of  the  client  machine. 


since  the  interface  is  similar  to  that  of  a  local  disk.  The  abstraction  provided  is  simply  that  of 
virtual  disk  pages.  Although  read-only  sharing  of  fles  is  simple  with  this  technique,  write  sharing 
poses  dHicultos. 

The  dbk  server's  interface  is  too  low-level  to  implement  concurrent  write  operations  property.  For 
example,  there  is  no  way  to  lock  a  file  or  to  enforce  access  controls.  Instead,  the  client  operating 
systems  have  to  negotiate  among  themselves  using  a  separate  protocol. 

An  intermediate  level  approach  is  to  provide  an  abstraction  of  files  with  unique  IDs.  The  interface 
to  such  a  file  server  can  alow  individual  blocks  of  fles  to  be  read  or  written,  as  well  as  logical 
operations  on  the  entire  file  such  as  locking.  Fie  servers  of  this  type  are  usually  accessed  via  a 
directory  system,  which  must  itself  be  a  shared  service. 

The  highest  level  approach  is  to  use  a  complete  file  and  directory  server,  functionally  equivalent 
to  the  fie  and  directory  system  on  a  dent  machine.  Interfacing  is  again  simple  because  file 
operations  can  be  intercepted  at  a  high  level  and  redacted  to  the  remote  server. 

123.  Integrating  Workstation  Disks  and  Fite  Sarvsra 

Another  issue  that  is  raised  when  workstations  are  networked  wth  file  servers  is  how  to  use 
workstation  disks  most  effectively.  One  successful  method,  used  in  the  Cedar  file  system  [32], 
considers  al  shared  files  to  be  immutable  (read-only),  and  uses  each  workstation  file  system  as  a 
cache  for  some  portion  of  the  globafiy  shared  Us  system.  Fles  are  created  on  the  local  file 
system  and  remain  private  untl  they  are  stored  beck  on  the  shared  fito  server.  From  that  time  on. 
that  version  of  the  He  may  not  be  modHed,  and  may  be  shared  by  other  users  (subject  to  normal 
protection  mechanisms,  of  course),  guaranteeing  consistency  Is  relatively  simple;  the  shared  file 
server  must  provide  atomic  creation  of  a  now  version  of  a  He. 

A  dfferent  approach  is  taken  by  Vie  designers  of  the  Camegie-Melon  ITC  file  system  [30]. 
Workstation  dsks  are  also  used  as  caches,  but  shared  fles  are  not  assumed  to  be  immutable. 
As  a  result,  cache  vaUaHon  is  required,  inlBatsd  ether  by  the  workstation  before  using  a  cached 
fito,  or  by  the  fie  server  when  a  shared  fito  is  modfied. 

124.  Integrating  Fite  System  Name  Spacas 

Once  He  servers  are  used  to  permit  sharing  of  files  in  a  network,  integration  of  many  file  name 
spaces  becomes  an  Issue.  The  Integrated  name  space  should  allow  a  fie  to  be  named  in  the 
same  way  from  any  machine  in  the  network,  in  order  to  foster  portable  programs  and  minimize 
confusion  when  users  change  workstations. 

If  the  (Afferent  fifo  servers  are  at  the  intermediate  or  low  level  described  above,  integration  can  be 
achieved  through  a  single  (logically  centralized)  directory  service.  A  more  common  case,  how¬ 
ever,  is  that  existing  workstations,  mainframes,  and  file  servers  all  have  their  own  file  and  direct¬ 
ory  systems  that  must  be  integrated  into  a  single  name  space.  For  tree-structured  name  spaces, 
two  schemes  are  possfoto.  The  first  uses  a  super-root  that  logically  contains  the  roots  of  all  file 
systems  in  the  network.  Addfi  tonal  syntax  is  used  to  refer  to  the  super-root  in  full  pathnames. 
For  example,  the  file  name  UMtwflb  might  be  used  to  refer  to  /usr/lib  on  machine  A  from  any 


other  machine  in  the  network.  Advantages  of  this  scheme  are  that  K  is  simple  to  implement  and 
guarantees  consistent  interpretation  of  tie  names  anywhere  in  the  network.  A  disadvantage  is 
that  this  approach  is  not  transparent  since  the  location  of  a  remote  file  is  reflected  in  its  full  path 
name.  This  problem  can  be  circumvented  through  the  use  of  symbolic  links,  a  directory  system 
feature  which  slows  a  user  to  impose  an  arbitrary  view  on  top  of  the  actual  tree  structure. 

The  second  scheme  slows  remote  mount  points  in  each  local  directory  tree,  so  that  each  direct¬ 
ory  system  may  have  a  different  view  of  the  dttributod  system.  The  problem  with  this  approach 
is  that  consistent  interpretation  of  names  must  be  obtained  by  convention;  I  is  not  enforced  by 
any  mechanism.  The  logic  of  name  interpretation  on  the  local  machine  is  also  more  complicated. 
On  the  other  hand,  I  gives  IndMdual  machines  more  control  over  their  view  of  the  name  space. 

125.  File  Servers  versus  Database  Servers 

There  is  growing  agreement  among  designers  of  (Sstrftxited  fie  systems  that  I  is  important  to 
dtetinguish  between  fie  system  and  database  system  functionally.  For  example,  file  servers 
must  support  efficient  sequential  reading  of  smal  files  and  creation  of  new  versions  of  flies,  but 
probably  do  not  need  to  support  large  lies  or  synchronized  modi icat ion  of  portions  of  files.  Data¬ 
base  servers,  on  the  other  hand,  can  be  used  for  transactional  updates  to  shared  information  and 
efficient  access  to  large  lies.  Making  this  dtotincUon  slows  optimized  file  server  and  database 
server  designs,  rather  than  compromised  designs  stretched  to  fit  both  classes  of  needs. 

13.  Fault  Tolerance 

Another  area  in  which  dtatributed  systems  diner  from  centralized  systems  is  failure  semantics. 
Partial  faNures,  in  which  some  but  not  afl  of  the  components  of  a  system  continue  to  function,  are 
more  common  in  dtetrtbuted  systems  and  add  to  their  complexity.  Various  mechanisms  are  used 
in  order  to  cope  with  this  complexity.  The  book  by  Anderson  and  Lee  presents  a  thorough 
overview  of  fault  tolerance  techniques  [2]. 

13.1.  Transactions  for  Reliability 

Transactions  are  used  to  simplify  the  construction  of  reliable  distributed  programs,  ones  which  do 
not  lose  or  corrupt  data.  Transactions  were  first  used  in  database  systems  [14],  but  have  since 
been  adopted  in  operating  systems  [33]  and  programming  languages  [20].  A  transaction  has 
three  essential  properties,  each  of  which  must  be  guaranteed  even  in  the  presence  of  processor 
and  communication  falures. 

Serfalzablity,  the  first  property,  means  that  the  concurrent  execution  of  any  number  of  trans¬ 
actions  is  equivalent  to  their  serial  execution  in  some  order.  This  property  insures  that  I  each 
transaction  transforms  a  consistent  database  state  kilo  another  consistent  database  state,  the 
overal  consistency  of  the  database  is  preserved  when  transactions  execute  concurrently. 

The  second  property  it  atomicity,  which  guarantees  that  a  transaction  is  an  all-or-nothing  opera¬ 
tion;  no  partial  effects  of  a  transaction  are  ever  vWbie  to  other  transactions.  When  more  than  one 
processor  is  involved,  this  requires  some  form  of  (fistributed  commit  protocol,  the  most  well 


known  of  which  to  two-phase  commit  [14, 19].  At  any  time  before  committing,  a  transaction  may 
abort,  leaving  the  system  state  as  V  the  transaction  had  never  been  executed.  The  fact  that 
intermeddle  effects  are  not  viable  to  other  transactions  means  that  the  domino  effect  (cascaded 
aborts)  cannot  occur.  When  a  transaction  is  aborted,  one  can  be  sure  that  no  other  transaction, 
either  stl  running  or  already  committed,  could  have  rated  on  updates  performed  by  the  aborted 
transaction. 

The  third  property  is  permanence,  which  states  that  once  a  transaction  commits,  Is  effects  be¬ 
come  permanent.  Providing  permanence  in  the  presence  of  failures  requires  some  form  of  stable 
storage  [19].  This  involves  writing  each  logical  page  of  data  onto  more  than  one  disk  and  modi¬ 
fying  the  read  and  crash  recovery  operations  to  take  advantage  of  the  redundancy.  It  is  still 
posstoie  that  the  copies  of  the  disk  page  can  become  corrupted  in  such  a  way  that  the  read 
operation  would  fal;  but  by  increasing  the  degree  of  replication,  the  probability  of  such  a  cata¬ 
strophic  failure  can  be  made  arblrarRy  small. 

Crash  recovery  mechanisms  use  stable  storage  in  hvo  ways:  for  checkpoints  and  logs.  A  check¬ 
point  to  a  snapshot  of  a  consistent  state  that  can  be  restored  after  a  crash.  A  log  is  a  record  of 
the  events  or  operations  that  affect  the  state  of  the  system;  I  to  replayed  after  a  crash.  Check¬ 
points  provide  faster  crash  recovery,  while  logs  are  less  expensive  during  normal  operation.  If  a 
combination  of  these  two  schemes  to  used,  the  log  need  only  be  replayed  from  the  most  recent 
checkpoint,  and  the  time  between  checkpoints  can  be  used  to  balance  the  cost  of  the  normal  and 
recovery  modes  of  operation. 

13.2.  Nested  Transactions 

Nested  transactions  are  a  generalization  of  single-level  atomic  transactions,  in  order  to  allow 
them  to  mesh  property  with  the  concepts  of  composition  and  abstraction  supported  by  program¬ 
ming  languages.  In  this  scheme,  a  transaction  consists  of  a  tree  of  subtransactions,  with  a  single 
top-level  transaction  at  the  root.  The  Mermedtoto  effects  of  a  transaction  that  has  not  yet  com¬ 
mitted  are  viable  only  to  Is  descendants  in  the  tree.  The  effects  of  a  committed  subtransaction 
are  viable  only  to  ancestors  and  stolngs  In  the  tree.  If  a  transaction  aborts,  any  uncommitted 
subtransactions  must  be  aborted,  and  the  effects  of  any  committed  subtransactions  must  be 
undone.  The  nested  transaction  model  was  chosen  tor  the  Argus  system  at  MIT  [20]. 

13.3.  Replication  tor  Availability 

The  availability  of  a  system  is  the  probability  that  the  system  will  be  up  (either  at  a  particular  time 
or  on  average).  Replication  is  used  to  increase  the  availability  of  dist touted  systems,  either 
through  the  use  of  a  primary/standby  archlecture  or  via  a  modular  redundancy  scheme,  in  a 
primary/standby  scheme,  only  a  single  component  performs  ks  normal  functions;  tel  the  other 
components  are  on  standby  in  case  the  primary  fate.  In  a  modular  redundancy  approach,  tel 
components  perform  the  same  function,  and  some  form  of  voting  on  the  outputs  to  used  to  mask 
failures. 

A  classic  primary/standby  architecture  to  Tandem’s  method  of  process  pairs  [4].  The  processes 
In  a  process  pair  execute  on  different  physical  processors.  One  process  to  designated  as  the 
primary,  the  other  as  the  standby.  Before  each  request  to  processed,  the  primary  sends  Wor- 
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(ration  about  Ns  Internal  state  to  the  standby  in  the  form  of  a  checkpoint.  The  checkpoint  enables 
the  standby  to  complete  the  request  V  the  primary  fails. 

The  Ms  project  at  Cornel  uses  a  primary/standby  archlecture  for  replcated  objects  [5].  in  each 
Interaction  wlh  a  roplcated  object  in  Isis,  one  replica  plays  the  role  of  ooordtoator,  and  only  it 
performa  the  operation.  The  coordhator  then  uses  a  two-phase  commit  protocol  to  update  the 
other  neplcas. 

Triple  modular  and  Nroodular  redundancy  have  long  been  familiar  to  designers  of  fault-tolerant 
computer  systems  [2].  in  triple  modular  redundancy,  every  computation  is  carried  out  by  each  of 
three  processors.  The  results  are  then  compared,  and  I  at  least  two  agree,  that  value  is  used,  in 
the  Circus  system,  replcation  was  integrated  wlh  remote  procedure  can  in  order  to  support  mod¬ 
ular  redundancy  at  the  program  module  level  19]. 

Gilford’s  weighted  voting  scheme  uses  quorums  and  version  numbers  to  provide  replication 
transparency  for  (Has  [13].  In  this  algorithm,  read  and  write  quorums  (eets  of  replcas)  are  chosen 
so  that  any  read  operation  wfll  include  the  most  recently  written  version.  Hertihy  extended 
Gilford’s  algorlhm  to  handle  replcated  abstract  data  types  [15].  m  Herltiy's  approach,  con¬ 
straints  on  quorum  assignments  are  derived  from  analysis  of  the  semantics  of  the  abstract  data 
types. 

14.  Conclusion 

Well  designed  distributed  systems  should  strike  appropriate  balances  between  the  needs  for 
integration  and  autonomy,  and  between  the  needs  for  Increased  performance  and  increased 
availability.  The  lemized  points  below  represent  the  features  that  project  members  recommend 
for  inclusion  in  the  operating  system,  programming  language,  and  support  environment  of  any 
future  distributed  system. 

•  Message-based  kernel 

•  Transparent  network  inter-process  communication 

•  Remote  procedure  call  fadlty 

•  Group  communication  integrated  wlh  remote  procedure  cal 

•  Conventional  aoftware  tools  extended  for  distributed  environments 

•  Lightweight  processes 

•  Distributed  tie  system 

•  Distributed  database  system 

•  Unix  compatttiky 
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